ISP: Large-Scale In-memory Spatial Data Processing System (Demo Paper)

Huge amount of spatial data such as GPS locations is being generated everyday, which brings big challenges of efficient spatial data processing. Many existing big spatial data processing techniques are mostly based on disk-resident systems. They have not fully taken advantages of modern hardware, such as large main memory capacities and multi-core processors. In this paper, we demonstrate our ISP system for in-memory processing of large-scale spatial data in distributed multi-core computing nodes. ISP is built on top of the open source Impala system, a leading Massively Parallel Processing (MPP) SQL engine, with two signficant extensions. First, while Impala is designed to process relational data and does not support spatial queries, ISP supports spatial SQL query syntax at the front end and is able to process the spatial queries at the back end. Second, while Impala currently supports neither indexed joins nor parallel joins on multi-core machines for non-equality joins, ISP provides on-the-fly parallel spatial indexing and query processing modules. We have performed experiments for a case study of point-in-polygon test based spatial joins. Using real data for a point-in-polygon based spatial join, experiments have shown that ISP on a two-node minicluster is 6.4X time faster than PostgreSQL/PostGIS. ISP is also 10X faster than Hadoop-GIS, a big spatial data processing solution built on top of Hadoop/Hive, on a 10-node Amazon EC2 cloud cluster. With proper setting of parameters of distributed systems, ISP also scales well. We will demonstrate ISP using both an EC2 cloud cluster and an in-house small cluster to conference participants.